Statistical Signal Processing
This page contains resources about Statistical Signal Processing, Point Estimation, Estimation Theory, Adaptive Filtering, Adaptive Signal Processing and Adaptive Filter Theory, System Identification and Adaptive Array Processing. Subfields and Concepts * Linear Shift-Invariant (LSI) filter / LTI filter * Filtering Problem * Adaptive filter ** Shift-Varying filter structure ** Criterion of performance ** Adaptive Algorithm * Four major configurations of an adaptive filter ** System Identification ** Noise Cancellation ** Prediction ** Inverse System Identification * Basic filter representations ** Transversal (direct form) filter structure ** Symmetric transversal filter structure ** Lattice filter structure ** Parallel form filter structure ** State-space representation ** Innovations representation / Innovations filter structure * FIR filter / Non-recursive adaptive filter ** Gradient Descent / Steepest Descent ** Normalized Nonlinear Gradient Descent (NNGD) ** Fully Adaptive NNGD (FANNGD) ** Unnormalized Gradient / Least Mean Squares (LMS) filter ** Normalized Gradient / Normalized LMS (NLMS) filter ** Leaky LMS filter ** Gradient Adaptive Lattice (GAL) filter ** Lattice LMS filter with Joint Process Estimation ** FIR Wiener filter * IIR filter / Recursive adaptive filter ** Recursive Least Squares (RLS) filter / Forgetting Factor ** Kernel RLS (KRLS) filter ** Sliding Window RLS filter ** IIR Wiener filter * Polynomial Regression filters * Gaussian Regression filter * Online State Estimation / Recursive Estimation ** Bayesian Recursive Estimation / Bayes filter ** Kalman filter ** Extended Kalman filter (EKF) ** Unscented Kalman filter (UKF) ** Iterated EKF ** Information filter ** Interacting Multiple Models (IMM) Filter ** Histogram filter ** Monte Carlo Methods (Approximation to Bayesian Estimation) *** Particle filter * Optimum filters ** Eigenfilter ** Kalman filter ** Wiener filter * Square Root filter / Cholesky Decomposition-based Kalman Filter ** Exponentially Weighted RLS filter ** QR Decomposition-based RLS (QRD-RLS) ** Extended QRD-RLS ** Inverse QRD-RLS * Hierarchical filters ** Hierarchical Gradient Descent (HGD) ** Hierarchical LMS (HLMS) filter * Complex valued filters ** Complex LMS (CLMS) filter * Time-domain Methods ** Autocorrelation Function ** Cross-Correlation ** Cross-Covariance ** Correlogram * Frequency-domain Methods / Spectral Estimation ** Nonparametric Methods *** Periodogram *** Modified Periodogram *** Barlett's Method / Periodogram Averaging *** Welch's Method / Modified Periodogram Averaging *** Blackman-Tukey Method / Periodogram Smoothing *** Multiple Window Method *** Empirical Transfer Function Estimation (ETFE) ** Minimum Variance Spectral Estimation (MVSE) ** Maximum Entropy Method (MEM) ** Parametric Methods *** AR Spectral Estimation (using All-Pole Transfer Function) *** MA Spectral Estimation (using All-Zero Transfer Function) *** ARMA Spectral Estimation (using Pole-Zero Transfer Function) *** Minimum Variance Distortionless Response (MVDR) filter *** Linearly Constrained Minimum Variance (LCMV) filter *** Subspace Methods ** Frequency Estimation (using a Harmonic Process / Sinusoid Model) *** Pisarenko Harmonic Decomposition *** Multiple Signal Classification (MUSIC) Pseudospectrum *** Eigenvector Method *** Pencil Method *** Frequency-domain version of Prony's Method *** Minimum Norm Algorithm *** Subspace Methods / Eigendecomposition-based Methods **** Blackman-Tukey Principal Components Method **** AR Method **** Minimum Variance Method ** High-resolution / Super-resolution Spectral Estimators *** MVDR *** LCMV *** MUSIC * Signal Modelling (mainly using All-Pole Transfer Funcion) ** Linear Nonparametric Signal Models *** Linear random signal model *** Recursive representation *** Innovations representation ** Parametric Pole-Zero Signal Models *** Autoregressive (AR) model / All-Pole model *** Moving Average (MA) model / All-Zero model *** ARMA model / Pole-Zero model ** Least Squares Method ** Padé Approximantion ** Prony's Method ** Levinson–Durbin Algorithm ** Lattice filter ** Iterative Prefiltering ** Final Data Records / Block Estimation / Data Windowing / Finite Interval Modelling *** Yule Walker Method / Autocorrelation Method *** Covariance Method *** Modified Covariance *** Pre-windowing Method *** Post-windowing Method *** Unbiased Autocorrelation Estimate *** Burg Method *** Forward and Backward Linear Prediction (FBLP) ** Stochastic Modelling *** Modified Yule-Walker Equation (MYWE) Method *** Least Squares MYWE Method *** MA Model using Spectral Factorization *** Durbin's Method * * Bayesian Linear Regression * Regularization ** Regularized least squares ** L1-regularization / LASSO ** L2-regularization / Ridge Regression * Iterative Methods ** Expectation-Maximization (EM) Algorithm ** Levenberg–Marquardt Algorithm ** Iteratively Reweighted Least Squares ** Nonlinear Least Squares ** Gradient Descent / Steepest Descent ** Krylov Subspace Methods *** Conjugate Gradient Method ** Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm * Risk Function / Expected Loss (i.e. Expectation Value of Loss Function) ** Mean Square Error (MSE) * Estimators ** Biased Estimator ** (Asymptotically) Unbiased Estimator ** Inconsistent Estimator ** (Asymptotically) Consistent Estimator * Frequentist Parameter Estimation ** Frequentist Risk * Bayesian Parameter Estimation / Bayesian Point Estimation / Bayesian Methods ** Posterior Risk / Bayesian Risk ** Bayesian Density Estimation ** Bayes filter / Recursive Bayesian Estimation *** Kalman filter ** Nonparametric Empirical Bayes (NPEB) ** Parametric Empirical Bayes Point Estimation ** Exponential family ** Conjugate prior * Methods for finding estimators ** Method of Moments / Moment Matching ** Maximum Likelihood Estimate (MLE) ** Maximum a posteriori (MAP) ** Pseudolikelihood ** Bayes Estimator / Bayesian Decision Theory *** Conjugate prior *** Exponential family *** Utility Theory ** Minimum-variance unbiased estimator (MVUE) ** Cramér–Rao bound / Cramér–Rao lower bound ** Median-unbiased estimator * Methods for evaluating estimators ** Minimum mean squared error (MMSE) / Bayes least squared error (BLSE) ** Minimum sum of squared errors ** Best linear unbiased estimator (BLUE) ** James–Stein estimator * M-estimators ** MLE ** Steepest Descent / Gradient Descent *** Stochastic Gradient Descent (SGD) *** Generalised Normalised Gradient Descent (GNGD) *** Hierarchical Gradient Descent (HGD) *** Normalized Nonlinear Gradient Descent (NNGD) *** Fully Adaptive NNGD (FANNGD) * Stochastic Optimization ** Stochastic Approximation * Sufficiency, Completeness and Variance Reduction Techniques (VRT) ** Factorization Theorem ** Minimal Sufficient Statistic ** Rao–Blackwell theorem *** Rao–Blackwellization *** Rao–Blackwell estimator ** Exponential family ** Conjugate prior family * Black-box and grey-box models * Time Series Models ** ARMA Models ** Volatility Models / Conditional Heteroscedastic Models / ARCH Models ** ARMA Metrics (e.g. Martin Distance) * Stochastic Systems ** Stochastic Processes ** Stochastic Calculus ** Stochastic Optimal Control ** Stochastic Time Series * Signal Detection Theory * The Expected Loss Principle * Adaptive Control ** Auto-tuning / Self-tuning * Adaptive Array Processing ** Adaptive beamforming ** Matched subspace filter / Matched subspace detector ** Angle estimation * Bayesian Information Theory ** Minimum Message Length (MML) ** Occam's Razor * Model Order Selection / Model Comparison ** Akaike Information Criterion (AIC) ** Bayesian Information Criterion (BIC) ** Minimum Description Lenght (MDL) ** Akaike Final Prediction Error (FPE) ** Parzen's Criterion AR Transfer Function (CAT) * Subspace Identification ** N4ASID ** MOESP ** CVA / CCA ** SSARX ** Frequency-domain subspace identification ** Time-domain subspace identification * Nonlinear Adaptive Filtering / Nonlinear System Identification ** Blind Deconvolution *** Unsupervised adaptive filter / Blind equalizer / Blind adaptive filter *** Bussgang Algorithm ** Artificial Neural Networks / Adaptive Neural filters *** Feedforward Neural Network *** Recurrent Neural Network ** NARMAX Models ** Volterra Series Models ** Block Structured Models *** Hammerstein systems *** Wiener systems / Parallel cascade nonlinear systems * Subspace Methods / Subspace Learning / Dimensionality Reduction ** Supervised Learning *** Feature Selection ** Unsupervised Learning *** Singular Vlaue Decomposition (SVD) *** Principal Components Analysis (PCA) ** Subspace Tracking *** Grassmannian Rank-One Update Subspace Estimation (GROUSE) *** Parallel Estimation and Tracking by REcursive Least Squares (PETRELS) *** Multiscale Online Union of Subspaces Estimation (MOUSSE) *** Grassmannian Robust Adaptive Subspace Tracking Algorithm (GRASTA) *** Online Supervised Dimensionality Reduction (OSDR) Online Courses Video Lectures * Adaptive filters by Ali H. Sayed Lecture Notes * Advanced Signal Processing by Danilo Mandic * Spectral Estimation and Adaptive Filtering by Danilo Mandic * Stochastic Systems by Florian Herzog * System Identification by Roy Smith * Recursive Estimation by Rafaello D'Andrea Books and Book Chapters * Candy, J. V. (2016). Bayesian signal processing: Classical, modern and particle filtering methods. 2nd Ed. John Wiley & Sons. * Box, G. E., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis: forecasting and control. John Wiley & Sons. * Kumar, P. R., & Varaiya, P. (2015). Stochastic systems: Estimation, identification, and adaptive control. SIAM. * Haykin , S. (2014). Adaptive filter theory. 5th Ed. Prentice Hall. * Tangirala, A. K. (2014). Principles of System Identification: Theory and Practice. CRC Press. * Goodwin, G. C., & Sin, K. S. (2014). Adaptive filtering prediction and control. Dover Publications. * Åström, K. J., & Wittenmark, B. (2013). Adaptive control. Dover Publications. * Van Trees, H. L. (2013). Detection Estimation and Modulation Theory, Part I: Detection, Estimation, and Filtering Theory. 2nd Ed. John Wiley & Sons. * Aster, R. C., Borchers, B., & Thurber, C. (2012) Parameter estimation and inverse problems. Academic Press * Diniz, P. S. (2012). Adaptive filtering. Springer . * Sayed, A. H. (2011). Adaptive filters. John Wiley & Sons. * Adali, T., & Haykin, S. (2010). Adaptive signal processing: next generation solutions. John Wiley & Sons. * Hayes, M. H. (2009). Statistical digital signal processing and modeling. John Wiley & Sons. * Mandic, D. P., & Goh, V. S. L. (2009). Complex valued nonlinear adaptive filters: noncircularity, widely linear and neural models (Vol. 59). John Wiley & Sons. * Kulkarni, V. G. (2009). Modeling and analysis of stochastic systems. 2nd Ed. CRC Press. * Porat, B. (2008). Digital processing of random signals: theory and methods. Prentice-Hall. * Vaseghi, S. V. (2008). Advanced digital signal processing and noise reduction. John Wiley & Sons. * Van den Bos, A. (2007). Parameter estimation for scientists and engineers. John Wiley & Sons. * Poularikas, A. D., & Ramadan, Z. M. (2006). Adaptive filtering primer with MATLAB. CRC Press. * Manolakis, D. G., Ingle, V. K., & Kogon, S. M. (2005). Statistical and adaptive signal processing: spectral estimation, signal modeling, adaptive filtering, and array processing. Norwood: Artech House. * Gray, R. M., & Davisson, L. D. (2004). An introduction to statistical signal processing. Cambridge University Press. * Lehmann, E. L., & Casella, G. (2003). Theory of point estimation. Springer. * Casella, G., & Berger, R. L. (2002). Statistical inference. Cengage Learning. * Cichocki, A., & Amari, S. I. (2002). Adaptive blind signal and image processing: learning algorithms and applications. John Wiley & Sons. * Nelles, O. (2001). Nonlinear system identification: from classical approaches to neural networks and fuzzy models. Springer Science & Business Media. * Moon, T. K. S., & Wynn, C. (2000). Mathematical methods and algorithms for signal processing. Pearson. * Ljung, L. (1999). System identification: theory for the user. PTR Prentice Hall, Upper Saddle River, NJ. * Bretthorst, G. L. (1998). Bayesian spectrum analysis and parameter estimation (Vol. 48). Springer Science & Business Media. * Van Overschee, P., & De Moor, B. L. (1996). Subspace identification for linear systems: Theory—Implementation—Applications. Springer. * Kay, S. M. (1993). Fundamentals of Statistical Signal Processing, Vol I: Estimation Theory. * Therrien, C. W. (1992). Discrete random signals and statistical signal processing. Prentice Hall PTR. * Scharf, L. L. (1991). Statistical signal processing (Vol. 98). Reading, MA: Addison-Wesley. * Kay, S. M. (1988). Modern spectral estimation. Pearson Education. * Marple, S. L. (1987). Digital Spectral Analysis. Prentice Hall. Scholarly Articles * Geering, H. P., Dondi, G., Herzog, F., & Keel, S. (2011). Stochastic systems. Course script. (link) * Vaidyanathan, P. P. (2007). The theory of linear prediction. Synthesis lectures on signal processing, 2''(1), 1-184. * de Jesús Rubio, J., & Yu, W. (2007). Nonlinear system identification with recurrent neural networks and dead-zone Kalman filter algorithm. ''Neurocomputing, 70(13), 2460-2466. * De Cock, K., De Moor, B., & Leuven, K. U. (2003). Subspace identification methods''.'' Control systems robotics and automation, Vol. 1 of 3, pp. 933-979 * Parlos, A. G., Menon, S. K., & Atiya, A. (2001). An algorithmic approach to adaptive state filtering using recurrent neural networks. IEEE Transactions on Neural Networks, 12(6), 1411-1432. * Griffiths, J. W. R. (1983). Adaptive array processing. A tutorial. Communications, Radar and Signal Processing, IEE Proceedings F, 130(1), 3. * Billings, S. A. (1980). Identification of nonlinear systems: a survey. In IEE Proceedings D-Control Theory and Applications (Vol. 127, No. 6, pp. 272-285). IET. Software * System Identification Toolbox - MATLAB * Adaptive Filters (Digital Signal Processing Toolbox) - MATLAB See also * Optimization * Optimal Control * Robust Control * Probabilistic Graphical Models Other Resources * Subspace Tracking by Laura Balzano * Course notes on Classical and Bayesian Statistics * AllSignalProcessing.Com - Lessons and MATLAB code * Filtering, State Estimation, and Other Forms of Signal Processing - Notebook Category:Signal Processing Category:Probability and Statistics Category:Control Theory Category:Machine Learning